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Purpose of the Survey 

• Need for highly trained professionals in educational 
measurement, assessment, and psychometrics 

- NCLB mandated testing and educational accountability 

— Increase in licensure and certification testing 

• Brennan & Plake (1991) and Patelis, Kolen, & Parshall (1997): 
academic and nonacademic organizations have problems 
recruiting; low supply of qualified candidates at entry- and 
mid-level 

• No studies have examined salary 

— APA, SIOP, ASA conduct these surveys 
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Overview 


Results from a 2007 survey of educational 
measurement professionals 

> Methodology 

> Demographics & Descriptive Statistics 
>Job Analysis 

> Compensation 
^Implications 
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Methodology: Survey 

• Online survey 

• 80 Questions covering a variety of areas: demographics, 
education, employment & compensation, job tasks, 
professional experiences, and memberships, conference 
attendance & presentations 

• Small pilot (n = 10) to check administration time and 
applicability across various jobs 

• Open from February 1 - April 17, 2007 

• Median completion time = 10.4 minutes 
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Methodology: Sample 

• Contacted members of: 

— National Council on Measurement in Education (NCME) 
— National Association of Test Directors (NATD) 

— APA Division 5 (Evaluation, Measurement, & Statistics) 

• Received an electronic or mailed invitation to 
participate with a link to the survey 

• 538 respondents 

• Anonymous 
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Job Setting 



Academic Setting 

Research or Testing Organization 

Other 

District, State, Federal Setting 

Missing 

Total 


# 

% 

200 

37.2 

200 

37.2 

57 

10.6 

51 

9.5 

30 

5.6 

538 

100.0 


0 
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Gender 



■ Male 

■ Female 


Total Academic District Testing Other 

(N=530) (n=199) (n=51) (n=196) (n=56) 
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100 % 


Race/Ethnicity 


90% 

80% 

70% 

60% 

50% 

40% 

30% 

20 % 

10 % 

0 % 



Total Academic District Testing Other 

(N=538) (n=200) (n=51) (n=200) (n=57) 


■ Caucasian (non-Hispanic) 

■ Asian, Asian American, Pacific 
Islander 

■ Prefer not to Indicate 

■ Other 

■ Hispanic or Latin American 

■ African American 
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Region 



Total Academic District Testing Other 

(N=538) (n=200) (n=51) (n=200) (n=57) 


■ Northeast 

■ Midwest 

■ South 

■ West 

■ Outside US 


0 
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100 % 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20 % 

10 % 

0 % 


4% 

20% 






18% 


23% 

6% 


Age Range 

Overall, Mean age = 49.5 (12.8) 


24% 



18% 


22% 


9% 



20% 


















12% 



18% 





16% 


31% 


19% 


28% 

4% 


7% 

26% 


32% 

9% 

19% 

7% 


Total (N=538) Academic District 

(n=200) (n=51) 


Testing Other (n=57) 
(n=200) 


70-86 

60-69 

50-59 

40-49 

30-39 

23-29 


0 
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Highest Degree Attained 

Overall, 15% currently enrolled in PhD program 



■ PhD 

■ MA/MS 

■ EdD 

■ MEd 

■ Other 

■ BA/BS 

■ MBA 

■ PsyD 


Total (N=538) Academic District (n=51) Testing (n=200) Other (n=57) 

(n=200) 


0 
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Years Since Highest Degree 


Overall, Mean years = 16.4 (12.3) 


100 % 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0 % 



Total (N=530) Academic District 

(n=200) (n=51) 


Testing Other (n=57) 
(n=200) 


30+ 

20-29 

10-19 

5-9 

0-4 
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Primary Job Tenure 

Overall, 83% work Full-Time (>30 hours) & 15% hold a second job 


100 % 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20 % 

10 % 

0 % 



■ > 15 years 

■ 10-15 years 

■ 5-10 years 

■ 2-5 years 

■ 1-2 years 

■ 6 months - 1 year 

■ < 6 months 


Total Academic District Testing Other 

(N=528) (n=200) (n=51) (n=200) (n=57) 
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■ College/Univ. Education 
Dept. 

■ College/Univ. Other Dept. 

■ College/Univ. Psychology 
Dept. 

■ College/Univ. Research 
Ctr./lnst. 

K-12 School System 

■ Prof. Licensing/Certification 
Org. 

Independent Consultant 

■ Other 
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District, State, Federal Setting (n=51) 

2 % 



K-12 School System 

■ State Department of 
Education 

District Department of 
Education 

■ Other 

■ College/Univ. Education Dept. 

■ Independent Consultant 

■ Federal Department of 
Education 
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1 % 


Research/Testing Org (n=200) 



Not-for-profit 

Research/Testing 
Commercial Research/Testing 

Prof. Licensing/Certification 
Org. 

College/Univ. Research 

Ctr./lnst. 

Consulting Firm 

College/Univ. Education Dept. 

Independent Consultant 

College/Univ. Other Dept. 

Commercial Textbook 

Publisher/Provider 

Other 
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Other (n=57) 



y CollegeBoard 


■ Independent Consultant 

■ Consulting Firm 

■ K-12 School System 

■ Other (please specify) 

■ College or University: Research 
Center / Institute 

■ Professional Licensing or 
Certification Organization 

■ College or University: Other 
Department 

■ College or University: Education 
Department 

Commercial Research or Testing 
Organization 

n Commercial Textbook Publisher 
/ Provider 

Not-for-profit Research or 
Testing Organization 
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Primary Job Telecommuting 



Total Academic District Testing 


(N=526) | 

[n=200) 

(n=51) 

(n=200) 


56% 


8% 


36% 


Other 

(n=57) 


■ 5 Days 


■ 3-4 Days 


■ 1-2 Days 


■ 0 Days 


■ Not offered by 
employer or for 
position 
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100 % 


Plans to Change Job 



Total (N=509) Academic District (n=51) Testing Other (n=57) 

(n=200) (n=200) 


No major change 
in the next 3 years 

■ Take a new job 
within field 

■ Retire 

■ Reduce to part- 
time or consultant 

■ Take a new job in 
different field 


0 
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Professional Experiences 

• 20 Yes/No questions about typical experiences within our 
field 

• More than 50% of respondents: 

- Developed, validated, or evaluated an instrument or the 
policies/procedures for utilizing an instrument 

- Participated in a research grant or contract 

- Conducted workshop training in education or 
measurement 

— Published an article in a journal 

— Served as an editorial consultant or reviewer in a journal 
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Professional Experiences 

• 50 - 20% of respondents: 

— Participated in a measurement workshop or formal continuing 
education activities in education or measurement 

- Taught graduate or undergraduate courses 

— Principal investigator or formally reviewed a research 
grant/contract 

— Appeared on radio/TV or within print media for education or 
measurement issues 

- Served as an editor, associate editor, or consulting editor in a 
journal 

• Less than 10% served as an expert witness in a court proceeding or 
served on a licensing or certification board 
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Professional Memberships 



# 

% 

National Council on Measurement in Education (NCME) 

422 

78.4 

American Educational Research Association (AERA) 

384 

71.4 


American Psychological Association (APA) 

120 

22.3 

Psychometric Society (PS) 

70 

13.0 

AERA - Regional Affiliate 

69 

12.8 

American Statistical Association (ASA) 

46 

8.6 

Association for Test Publishers (ATP) 

46 

8.6 

Association for Psychological Science (APS) 

36 

6.7 

National Association of Test Directors (NATD) 

35 

6.5 

Council of Chief State School Officers (CCSSO) 

26 

4.8 

Society for Industrial and Organizational Psychology (SIOP) 

19 

3.5 

International Association for Educational Assessment (IAEA) 

5 

0.9 
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Conference Attendance /Presentations 



Attended 
N (%) 

Presented last 3 yrs 
N (%) 

American Educational Research Association (AERA) 

259 

48 

192 

36 

National Council on Measurement in Education (NCME) 

242 

45 

143 

27 


Council of Chief State School Officers (CCSSO) 

52 

10 

53 

10 

AERA - Regional Affiliate 

42 

8 

52 

10 

Association for Test Publishers (ATP) 

38 

7 

38 

7 

American Psychological Association (APA) 

28 

5 

42 

8 

National Association of Test Directors (NATD) 

21 

4 

11 

2 

Psychometric Society (PS) 

19 

4 

23 

4 

Society for Industrial & Organizational Psychology (SIOP) 

15 

3 

12 

2 

Association for Psychological Science (APS) 

5 

1 

11 

2 

American Statistical Association (ASA) 

5 

1 

6 

1 

International Association for Educational Assessment (IAEA) 

2 

0 

6 

1 


0 
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Part II: 

Where We Work & What We Do 
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Data Sources 


(1) 5-point Likert-type scale on how time is spent 
in typical work week on specific tasks 

• 1 = Very little time, 5 = Most of my time 

• Rank order of tasks based on frequency of 
response in categories 4 + 5 
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The following questions pertain to specific activities you do at your primary job during a TYPICAL WORK WEEK. First, you 
will be asked to rate how much of your time is spent doing specific tasks within four categories. Second, you will be asked 
to approximate the percentage of time you spend within each category as a whole. 


* 39. Please indicate how much of your time is spent doing the following tasks, which involve TEACHING & 
WORKING WITH STUDENTS, where l=very little of my time and 5=most of my time 


Course Planning and/or 

1 

2 

3 

4 

5 

Mot Applicable 

Evaluation 

J 

J 

J 

J 

J 

J 

Dissertation and/or 

Thesis Mentoring 
Student Advisement 

j 

J 

J 

J 

J 

J 

and/or Evaluation 

J 

J 

J 

J 

J 

J 

Supervising Instructors 

or Teaching Assistants 

j 

J 


sj 

J 

j 

Teaching/Lecturing 

J 

J 

J 

J 

J 

J 

40. Please indicate how much of your time is spent doing the following tasks, which involve CONDUCTING 
RESEARCH, where l=very little of my time and 5=most of my time 


1 

2 

3 

4 

5 

Mot Applicable 

Data Collection 

J 

J 

J 

J 

J 

J 

Data Analysis 

J 

J 

J 

J 

J 

J 

Planning/Designing 

Research Study 
Proposal or Manuscript 
Preparation for 

j 

J 

J 

J 

J 

J 

Presentation or 

J 

J 

J 

J 

J 

J 


Data Sources 


(2) Percentage of time spent in typical work 
week in four categories of tasks 

| Teaching & Working with Students 
r Conducting Research 
r Professional Development & Service 
Administrative Tasks 

Other (additional tasks not included in the above categories) 
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Job Setting 


% 


# 


Academic Setting 

200 

37.2 

Research or Testing Organization 

200 

37.2 

Other 

57 

10.6 

District, State, or Federal Setting 

51 

9.5 

Missing 

30 

5.6 

Total 

538 

100.0 


a 
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Academic Setting 
Primary Job Titles 
n = 200 


% 


Assist, Assoc, Full, or Emeritus Prof 

68.0 

Graduate Student/TA 

10.5 

Adjunct Instructor 

8.5 

Administration/Director/Dean 

8.0 

Other 

4.0 


0 
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Academic 



Conducting Research 

Teaching & Working with Students 

Administrative Tasks 

Professional Development & 
Service 

Other 


n 

Average % 

184 

33.2 

183 

31.3 

177 

20.0 

175 

12.1 

105 

10.7 


0 
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Academic (n = 186) 


Conducting Research (33%) 

- Data Analysis (37%) 

- Preparing Manuscripts for Presentation/Publications (37%) 

Teaching (31%) 

- Teaching/Lecturing (28%) 

Admin Tasks (20%) 

- Attending Meetings (22%) 

Professional Development/Service (12%) 

- Editing/Reviewing Proposals/Manuscripts (18%) 
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Testing/Research Organization 
Primary Job Titles 
n = 200 



V P/ED/Dir of Psychometrics, Research or TD 


28.5 


Research Scientist 
Psychometrician 
Research Associate/Analyst 
Program Manager 
Other 

CEO or President 
Test Developer 


17.0 

16.0 

12.5 

10.5 

6.5 

5.5 

3.5 


0 
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Testing/Research Organization 


n Average % 


Operational Testing Procedures 189 30.9 

Management/Admin 186 27.7 

Conducting Research 191 27.5 

Professional Development 181 10.3 

Other 134 9.7 


0 
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Testing/Research Org (n = 193) 

Operational Testing Procedures (31%) 

- Psychometrics (44%) 

Management/Admin (28%) 

- Project management (55%) 

Conducting Research (28%) 

- Data analysis (40%) 

Professional Development (10%) 

- Conference attendance/presentation (16%) 
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District, State, Federal 
Primary Job Titles 
n = 51 



Director or Executive Director 


Research Specialist or Coordinator 


Other 


Consultant 


Assessment Specialist or Coordinator 


Psychometrician 


Psychologist 


0 
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District, State, 



Assessment/Accou nta bil ity 
Policy 

Conducting Research 


Management/Admin 

Other 


0 


CollegeBoard 


Federal 


n 


50 

49 

50 

49 

37 


Average % 


32.2 

19.8 

19.9 

17.6 

15.2 
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District, State, Federal (n = 50) 

Assessment/Accountabilitv (32%) 

- Managing Operational Testing Program (40%) 

Policy (20%) 

- Accountability Requirements (50%) 

- Evaluating Policy related to Accountability (48%) 
Conducting Research (20%) 

- Data analysis (50%) 

Management/Admin (18%) 

- Consultation w/ Stakeholders (46%) 
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Next Steps 


Additional analyses 

• Relationship of time-on-tasks with interesting 
variables... 

- Salary 

— Plans for changing jobs or retirement, etc. 

- Degree 

- Years since degree 
Future research 

• Are graduate programs providing appropriate training? 

• What training is needed for new professionals? Not-so- 
new professionals? 
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Part III: 

Show Me the Money 
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Does our profession need a compensation 

survey? 

• Benchmark compensation by setting, experience, 
degree and region 

• Inform prospective students and graduates 

• Inform organizations who recruit and retain testing 
professionals 

• Other related professions monitoring hiring and 
compensation practices every 3-5 yrs: 

- APA, SIOP, ASA, AFT, AAMC, Chronicle of Higher Ed 
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Compensation Survey 

• Cooperation from NCME and NATD, but not officially 
sanctioned. 

• Concern about competitive advantages among at least 
one corporate entity 

• Confidentiality - individuals, organizations - not much 
different than test score issues (student, school). 

• Benefits to field (profession, data based decisions) 
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Sources of Income 


• Primary Occupation 

— Academics (primarily faculty, some centers) 

- Research/Testing Organization (profit/non-profit, 
sponsor) 

- State/District/School/US DOE (and other gov.) 

- Other (private consultants, human services, other) 

• Secondary Occupation (same 4 settings) 

• Consulting 

• Technical (or Research) Advisory Panels 
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Primary Job Setting 


re 9e a rch - 1 n=200 $99,000 


othe 


m issi ng 


district 


academ i 


n=57 $75,614 


i n=1 1 $98,182 


i n=51 $88,039 


n=200 $75,525 


1 1 1 

$0 $ 25,000 $ 50,000 $ 75,000 


Bars show Mean 




$ 100,000 


Median Primary Income 



Median Income 


$ 300 , 000 - 

$ 275 , 000 - 


$ 240,00 

$ 220,00 

$ 200,00 

$ 180,00 

$ 160,00 

$ 140,00 

$ 120,00 

$ 100,00 

$ 80,00 

$ 60,00 

$ 40,00 




Doctoral 


M aster's 


Other 


Highest Degree Obtained 


Sex 

□ Female 

□ l prefer not to indicate 

□ Male 


Outliers are hidden 
Extreme values are hidden 


Salary for Primary Job x Highest Degree and Gender 






highest degree 

f~| Doctoral 

fl Master's 
| | Other 

Outliers are hidden 
Extreme values are hidden 


Salary for Primary Job x Work Setting 


Median Primary Income 


$240 

$220 

$200 

$180 

$160 

$140 

$120 

$100 

$80 

$60 

$40 

$0 


o 

o o 



o 


* 

* 


* 

* 

* 


o o 



Administration Ass't. Professor 

Assoc. Professor 


M issi ng 


Other 


Professor 

Prof. Emeritus 


Salary for Academics by Job Title 


Median Primary Income 


$240 

$220 

$200 

$180 

$160 

$140 

$120 

$100 

$80 

$60 

$40 

$0 


Salary for Primary Job x Yrs Since Doctoral Degree 


Outliers are hidden Doctorate Degree 

Extreme values are hidden | | Doctorate 



1 1 1 1 1 

0-9 years 10-19 years 20-29 years 30 plus years Missing 


Pearson correlations for Yrs. Since Degree, 
Yrs in Current Position and Salary from 
Primary Position(*p<.oi) 



Yrs Since Degree 
(N=525) 

Yrs in Current 
Position (N=528) 

Male 

343* 

181* 

Female 

498* 

321* 

Doctorate 

348* 

167* 

Masters 

491* 

323* 

Academic 

565* 

363* 

Res/Testing Org 

410* 

277* 

DOE 

494* 

205* 

Total 

421* 

260* 


35.00 i 



0 - 999 1,000 - 5,000 - 10,000 - 20,000 - 30,000 + I prefer not 

4,999 9,999 19,999 29,999 to indicate 


annual bonus 


Bonus from Primary Job (n = 98) 



Percent Consulting, Serving on TACs and Holding a Second 

Position x Work Setting 



□ Consulting 
■ TACs 

□ Second Positon 



Less $5,000 - $10,000 - $20,000 - $30,000 - $40,000 - $50,000 - $100,000 I prefer 
than 9,999 19,999 29,999 39,999 49,999 99,999 or not to 

$4,999 greater indicate 


Income from consulting, TACs, etc 


Income from TACs and Independent Consulting (n=247) 



Next steps 


• Additional analyses 

— Compensation by region 
— Compensation by location (proximity to Ig city) 

— Competencies 
— Significance tests 

• Submit manuscript to NCME for web (monograph with 
descriptives, cross tabs, table data) No access to individual data to 
prevent any identification of individuals/organizations. 

• Submit 2 papers to EMIP (compensation, job analysis) 

• Propose comprehensive compensation and job analysis practices 
survey to NCME Board with member directory 
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For more information, 
e-mail spackman@collegeboard.org 
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